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Introduction 

Robotic  tele-surgery  is  a  promising  application  of  robotics  to  medicine,  aiming  to 
enhance  the  dexterity  and  sensation  of  minimally  invasive  surgery  through  use  of  millimeter- 
scale  robotic  manipulators  under  control  of  the  surgeon.  Current  interfaces  do  not  enable  the 
surgeon  to  fully  exploit  his  capabilities  and  may  lead  to  underutilization  of  potential  capacities 
offered  by  robotic  tele-surgery  in  the  future.  In  an  attempt  to  advance  the  state  of  the  art  in  the 
field  of  human-machine  interface  (HMI)  and  of  the  properties  of  sensory  substitution,  the  Florida 
Institute  for  Human  and  Machine  Cognition  (IHMC)  initiated  a  new  project  called  Anthro-Centric 
Multisensory  Interface  for  Sensory  Augmentation  of  Tele-Surgery  (ACMI-SATS).  Its  purpose  is 
to  improve  the  effectiveness  of  tele-surgery  in  military,  as  well  as  civilian  applications  by 
improving  the  sensory  experience  and  the  control  input  system  using  HMI  principles  to  reduce 
surgeon  cognitive  workload  as  well  as  surgical  errors. 

Body 

The  use  of  teleoperated  surgical  (tele-surgical)  laparoscopic  robotic  systems  is  rapidly 
becoming  the  standard  of  practice  for  a  number  of  surgical  procedures.  This  technology  reduces 
surgical  trauma  to  the  patient,  lowers  postoperative  complications,  improves  surgeon 
ergonomics,  reduces  effects  of  surgeon  motion  tremors  and  enables  the  performance  of  novel 
techniques  (e.g.,  the  end  effector  can  perform  actions  that  a  human  appendage  cannot;  such  as 
locking  in  position  or  rotating  through  more  than  360  degrees).  These  “surgery  by  wire”  systems 
can  even  remove  the  need  to  transport  the  patient  to  the  surgeon’s  location.  Hirshauer  (1991), 
however,  notes  that  endoscopy  separates  the  eye  from  the  hand  of  the  surgeon,  and  this 
isolation  prevents  the  surgeon  from  perceiving  useful  kinesthetic  feedback  from  direct 
interaction  with  the  tissues  present  in  traditional  “open”  procedures.  Additionally,  the  reliance  on 
video  displays  creates  a  visual  information  flow  bottleneck.  This  increases  the  visual  workload, 
which  can  lead  to  increased  surgical  errors,  longer  intraoperative  sessions  (with  accompanying 
increased  risks  from  anesthesia)  and  decreased  number  of  procedures  performed  in  a  given 
time  period.  Technological  systems  that  utilize  visually  dominant  displays  such  as  those  used  in 
virtual  environment  (VE)  simulation  and  training  have  shown  improved  user  task  performance 
when  tactile  sensory  information  is  combined  with  visual  information  (Briggs  &  Srinivisan,  2001; 
He  &  Agah,  2001).  Virtual  reality  (VR)  training  systems  have  been  developed  to  bring  VE 
simulation  to  surgical  training  (Satava,  1995;  Gallagher  et  al.,  2005).  Kinesthetic  and  haptic 
signals  in  surgical  applications  are  critical,  and  prior  work  with  VEs  has  shown  that  errors 
increase  without  realistic  multi-sensory  interactions  (Cohn  et  al.,  2000).  Conversely,  combining 
multiple  sensory  modalities  (visual,  spatial  audio  and  tactile)  has  been  shown  to  improve 
performance  and  situation  awareness  in  complex  dynamic  tasks  (Raj,  Kass  &  Perry,  2000; 
Diamond  et  al.,  2003;  Olson  et  al.,  2004;  Erikson  et  al.,  2008;  Fuchs  et  al.,  2008;  Merlo,  Gilson 
&  Hancock,  2008). 

Telerobotic  systems  such  as  the  da  Vinci®  S  HD  (Intuitive  Surgical,  Sunnyvale,  CA) 
already  must  employ  torque,  position,  velocity  and  strain  sensors  used  to  maintain  accurate 
closed-loop  servomotion.  Beyond  control  loop  error  based  force-feedback,  however, 
presentation  of  these  data  streams  in  raw  form  would  not  provide  the  requisite  sensory  data 
needed  to  enhance  the  surgeon’s  experience.  The  addition  of  high  definition  stereoscopic  visual 
displays  can  provide  sufficient  cues  to  create  a  visual  perception  of  haptic  feedback  (Hagen  et 
al.,  2008).  The  sensory  experience,  however,  still  lacks  the  richness  and  subtleties  available 
during  open  procedures  and  decrements  procedural  consistency,  workload  and  operative 
efficiency  (Tavakoli,  Patel  &  Moallem,  2005;  Zhou  et  al.,  2008).  Moreover,  these  systems 
require  that  the  surgeon  remain  seated,  head  down  at  a  control  console  to  perceive  the  three 
dimensional  video  and  use  the  fixed  hand  controllers.  While  this  compares  favorably  to  standing 
astride  an  operating  table  using  laparoscopic  instruments,  the  most  favorable  ergonomics  would 
remove  all  restricted  motion,  posture  and  positioning  of  the  surgeon.  This  would  allow 
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telerobotic  surgeon  to  move  freely  during  a  procedure  to  reduce  fatigue  and  potentially  perform 
more  procedures  in  a  given  time  frame. 

In  the  military  medicine  domain,  tele-surgical  systems  have  the  potential  to  revolutionize 
combat  casualty  care  by  bringing  definitive  surgical  trauma  care  to  the  wounded  servicemember 
without  placing  surgical  specialists  in  harm’s  way.  Current  technology  developments  such  as  the 
United  States  Department  of  Defense  (DoD)  Defense  Advanced  Research  Projects  Agency 
(DARPA)  Trauma  Pod  (Figure  1)  seek  to  further  reduce  the  time  to  delivery  of  surgical 
intervention  by  installing  a  complete  tele-surgical  system  into  robotic  vehicles  (Burnett,  2007; 
Satava,  2005a,  2005b).  These  Trauma  Pods  could  be  remotely  driven  to  wounded  soldiers  in 
the  field,  providing  an  opportunity  to  treat  severe  wounds  within  the  “golden  hour”  following 
injury  (Pueschel,  2006).  Deploying  a  fleet  of  such  vehicles  to  a  theater  of  operations  would 
provide  maximum  benefit  by  providing  the  potential  for  immediate  care  to  multiple  wounded 
servicemembers  with  polytraumatic  injuries  following,  for  example,  a  detonation  of  an 
improvised  explosive  device  (IED).  In  such  a  scenario,  a  small  team  of  surgical  specialists  (e.g., 
orthopedic,  cardiovascular,  neuro,  general/trauma  surgeons,  etc.)  could  stabilize,  and  possibly 
provide  definitive  care  to,  the  maximum  number  of  casualties  if  they  could  easily  and  safely 
handoff  control  of  the  robotic  system  between  each  other  as  needed.  Because  there  would  be 
no  cross  contamination  between  patients  (or  need  to  scrub  or  change  sterile  garments),  the 
trauma  team  could  manage  the  surgical  needs  of  multiple  patients  more  effectively  that  in 
current  practice.  Experience  with  other  robotic  systems,  such  as  unmanned  aerial  vehicle 
(UAVs),  however,  has  shown  that  the  addition  of  robotic  technology  to  a  given  mission  can 
actually  adversely  increase  manpower  requirements. 


Figure  1:  DARPA  Trauma  Pod  Concept  (lead  by  SRI  International)  loading  a  wounded  servicemember 
(left),  evaluating  the  trauma  (center)  and  performing  surgery  (right).  Images  downloaded  from: 
http://www.technovelav.com/ct/Science-Fiction-News.asp?NewsNum=364.  and 

http://brl.ee.washinaton.edu/Research  Active/Suraerv/Proiect  09/Proiect  09.html 

The  addition  of  high  definition  (HD)  stereoscopic  visual  displays  to  robotic  surgery 
systems  provides  an  initial  step  toward  improved  sensory  interface  design.  However,  lack  of 
haptic  feedback  in  endoscopy,  can  lead  to  surgical  errors  such  as  tearing  of  tissues,  broken 
sutures  and  hemorrhage,  especially  in  delicate  or  intricate  procedures  (Bethea  et  al.,  2004; 
Vassiliades,  2006).  One  approach  to  enhancing  haptic  feedback  embeds  tactile  force  reflection 
transducers  on  the  fingertips  in  the  surgeon  console  hand  controllers  (Figure  2)  to  provide  a 
reflection  of  the  force  applied  by  endoscopic  graspers  capabilities  (Morimoto  et  al.,  1997;  Bar- 
Cohen  et  al.,  2001;  Tavakoli,  Patel,  &  Moallam,  2005;  Culjat  et  al.,  2008).  While  this  technique 
has  demonstrated  improved  surgical  technique  (reduced  grasping  force),  the  surgeon’s  auditory 
and  tactile  sensory  capabilities  remain  under  utilized  while  his  or  her  visual  sense  remains 
highly  task  loaded.  Alternate  haptic  interfaces  that  provide  end  effector  force  feedback  indicate 
resistance  to  commanded  movements  proportionally  and  complement  the  visual  display  to 
reduce  workload  (Wagner,  Stylopulos,  Jackson  &  Howe,  2007).  However,  the  narrow  field  of 
view  provided  by  the  endoscope  negatively  impacts  situation  awareness  (SA)  by  limiting 
perception  of  complex  spatial  relationships  within  the  body  cavity.  This  loss  of  SA  may  lead  to 
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inadvertent  incursion  into  sensitive  structures  following  camera  orientation  changes  or 
movement  of  internal  organs.  These  haptic  approaches  borrow  concepts  from  the  VE  and  VR 
worlds  that  place  a  high  premium  on  presence  (the  convincing  sense  of  perceiving  as  if 
physically  in  the  VR  or  VE).  In  contrast,  tele-surgery  must  place  the  highest  premium  on 
effective  treatment  of  the  patient  by  optimizing  perception  of  patient  state  to  the  surgeon,  rather 
than  presence.  Because  the  feedback  generated  by  the  sensors  in  the  operative  field  is  no 
longer  directly  coupled  to  the  surgeon’s  hands  (either  through  direct  contact  with  the  tissues  or 
indirect  contact  via  rigid  laparoscopic  instruments),  opportunities  now  exist  to  utilize  other 
sensory  substrates.  In  fact,  some  actions  in  surgical  robotic  manipulation  can  cause  the  surgeon 
to  make  inadvertent  hand  movements  due  to  haptic  force  feedback  errors  (Tavakoli,  Patel,  & 
Moallem,  2005).  Unlike  classic  VE/VR  approaches  to  haptics  in  tele-surgery,  these  alternative 
display  technologies  could  complement  or  supplant  traditional  haptics  without  affecting  or 
interfering  with  the  motion  of  the  primary  control  input  mechanism.  For  this  discussion,  haptics 
will  refer  to  telepresence  related  kinesthetic  sensory  interactions  while  tactile  will  refer  to  more 
general  or  abstract  tactual  interactions. 


Figure  2:  (left)  Electro-rheological  fluid  (ERF)  based  haptic  surgeon  hand  interface  (Bar-Cohen  et  al., 
2001);  (center)  Force  transducer  fitted  to  endoscopic  grasper;  (right)  Pneumatic  haptic  finger  tip  display 
(Culjat  et  al.,  2008)  integrated  with  da  Vinci®  hand  controllers. 

Perception  takes  place  in  the  brain,  not  at  the  end  organ  (Bach-y-Rita,  1972);  therefore, 
the  brain  can  learn  to  reinterpret  the  meaning  of  signals  from  specific  nerves  (e.g.,  from  tactile 
receptors)  given  appropriate  self-generated  feedback.  This  forms  the  basis  for  interfaces  that 
can  non-invasively  and  unobtrusively  use  alternative  sensory  pathways  to  provide  information. 
This  means  that  the  information  displayed  does  not  necessarily  need  to  represent  the 
underlying  data  at  high  resolution,  rather  abstract  representations  of  the  sensory  environment 
information  can  provide  sufficient  data  for  operator  decision  making  and  improved  SA  (Raj, 
Kass  &  Perry,  2000).  Such  sensory  substitution  mechanisms  have  been  demonstrated  in  tele- 
surgical  applications  (Kitagawa  et  al.,  2005)  and  exploit  the  plasticity  inherent  to  the  brain  and 
nervous  system,  which  supports  both  long  term  and  short  term  anatomical  and  functional 
remapping  of  sensory  data  (Finkel,  1990;  Walcott  &  Langdon,  2001).  Therefore,  a  new  class  of 
interfaces  could  be  incorporated  into  the  tele-surgical  surgeon  interface.  Using  alternative  tactile 
and  spatial  audio  displays  that  provide  higher  level  representations  of  data  could  provide 
metadata  information  or  data  that  a  surgeon  could  not  physically  sense  directly  even  when 
performing  an  open  procedure. 

Divorcing  the  surgeon’s  sensory  perception  from  direct  mechanical  linkage  to  the 
patient,  the  operative  field,  and  the  surgical  instruments  could  also  enable  a  paradigm  shift  in 
tele-surgical  methods.  Laparoscopic  instruments  were  initially  developed  as  extended  versions 
of  their  counterparts  used  in  open  procedures  (e.g.,  extended  length  scissors,  needle  drivers, 
etc.);  even  though  the  scaling  induced  poor  ergonomics,  familiarity  with  the  older  tools  existed. 
Likewise,  current  tele-surgery  systems  utilize  tools  that  mimic  the  laparoscopic  instruments, 
replacing  the  direct  mechanical  connection  with  an  electronic  one.  The  functional  capabilities 
(e.g.,  degrees  of  freedom)  of  the  robot  end  effectors  (such  as  the  da  Vinci®  EndoWrist®)  exceed 
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those  of  laparoscopic  instruments  and  human  hands.  Advances  in  head  mounted  displays  and 
motion  capture  from  the  commercial  sector  can  now  allow  both  freedom  of  movement  and  high 
resolution  stereoscopic  visual  perception  without  confining  the  user  to  a  fixed  console.  Motion 
capture  could  translate  movements  that  match  the  scale  and  trajectories  of  normal  (real-world) 
kinesthetic  interactions,  as  well  as  enable  the  use  of  motion  gestures  to  accomplish  supervisory- 
controlled  actions  to  take  advantage  of  the  higher  capabilities  of  modern  robotic  surgical 
systems.  Because  kinesthetic  perception  includes  both  self-perception  (proprioception)  and 
perception  of  changes  in  the  environment  (Hannford  &  Venema,  1995),  proprioceptive 
sensations  could  disambiguate  abstract  sensory  substitution  displays  of  end  effector  motion. 
The  full  use  of  arm  movements  (versus  hands  only)  could  provide  more  accurate  mapping  to  the 
degrees  of  freedom  available  in  modern  surgical  robots. 


Clearly,  adding  additional 
sensory  channels  would  increase 
the  volume  of  information  flow  to  the 
surgeon.  The  human  brain, 
however,  has  developed  to  make 
use  of  multi-sensory  inputs  in  the 
natural  world,  which  provides  a 
richer  sensory  environment  than 
most  man-made  systems.  This 
increase  information  transduction 
will  also  markedly  increase  the 
bandwidth  requirements  for  data 
flow  between  the  robot  (e.g., 
installed  in  a  Trauma  Pod)  and  the 
remote  surgeon  (Hannford  & 
Venema,  1995).  Recent  DoD 
projects  such  as  the  High  Altitude 
Platforms  Mobile  Robotic  Tele¬ 
surgery  (HAPsMRT)  (Broderick, 
2006;  Rosen  &  Hannaford,  2006) 
and  the  Smart  Codec  with  Tele¬ 
surgery  Capability  (Energid 
Technologies,  Cambridge,  MA), 
respectively,  have  developed 
methods  that  use  unmanned  aerial 
vehicles  to  reduce  the  radio 
frequency  signal  lag  between  the 
surgeon  and  the  robotic  end 
effectors  and  to  develop  efficient 
algorithms  for  transmission  of  data 
(Figure  3). 


Figure  3:  High  Altitude  Platforms  Mobile  Robotic  Tele¬ 
surgery  (HAPsMRT)  technology  demonstration  using 
real  time  data  compression  and  a  UAV  data  relay  to 
provide  haptic  interactions  in  tele-surgery  (Broderick, 
2006).  Surgeon  movements  (left,  lower)  are  relayed  via 
UAV  (left,  upper)  to  a  surgical  robot  (right,  upper)  with 
force  feedback  provided  through  haptic  enabled  hand 
controllers  to  accomplish  suture  task  (right,  lower). 


When  processing  and  inter-preting  tactile  data,  specific  traits  of  human  sensation  and 
cognitive  processing  such  as  adaptation,  habituation,  or  satiation  to  durative  stimuli  may 
interfere  with  perception  of  persistent  tactile  and  other  sensory  stimuli.  Adaptation  occurs  when 
a  signal  persists  for  many  minutes  (e.g.,  the  tactile  sensation  of  wearing  a  watch  is  automatically 
filtered  out  by  the  peripheral  and  central  nervous  system).  Habituation  occurs  when  a  durative 
signal  repeats  periodically  (e.g.,  a  ticking  clock)  and  is  no  longer  perceived.  Satiation,  also  the 
product  of  prolonged  stimulation,  produces  specific  spatial  distortions  (Cholewiak,  1976), 
perhaps  based  on  transient  plasticity  (Whitsel  et  al.,  1989;  Merzenich,  Recanzone,  Jenkins, 
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Allard  &  Nudo,  1988).  These  are  known  issues  with  known  solutions  currently  implemented  in 
IHMC's  Adaptive  Multiagent  Integration  (AMI)  software  (e.g.,  modulation  of  waveform 
characteristics  of  persistent  cues)  that  mitigate  their  effects.  AMI  agents  can  also  generate 
sensory  illusions,  such  as  spatial  audio,  the  saltation  rabbit  illusion  (Geldard  &  Sherrick,  1972, 
Geldard,  1975;  Bremer,  Pittenger,  Warren,  &  Jenkins,  1977)  and  spatial  summation  (Craig, 
1968).  Recent  studies  have  confirmed  that  tactile  illusions  are  processed  at  the  cortical  level  as 
if  they  were  real;  that  is,  no  differences  between  illusory  and  veridical  tactile  stimulation  patterns 
manifest  in  functional  magnetic  resonance  (fMRI)  scans  of  the  human  brain  (Blankenburg,  Ruff, 
Deichmann,  Rees  &  Driver,  2006).  Therefore,  tactile  illusions  could  improve  the  perception  of  a 
tactile  signal  beyond  the  physical  resolution  a  given  display. 

Prior  research  by  this  team  and  others  with  tactile  sensory  substitution  has  shown  that 
the  brain  is  capable  of  integrating  the  information  from  an  artificial  receptor,  arriving  via  a  brain- 
machine  interface  (BMI),  in  a  perceptual  experience  that  depends  upon  the  nature  of  the 
information  gathered  by  the  specific  artificial  receptor  (Bach-y-Rita,  1972,  1995;  Bach-y-Rita, 
Collins,  Saunders,  White  &  Scadden,  1969).  The  BrainPort®  device  (Wicab,  Inc.,  Madison,  Wl). 
provides  information  to  the  brain  through  electrotactile  stimulation  of  the  tongue  (Bach-y-Rita, 
Kaczmarek,  Tyler,  &  Garcia-Lara,  1998).  Receptors  on  the  tongue  surface  are  uniquely  qualified 
to  receive  electrical  impulses  because  of  the  density  and  sensitivity  of  nerve  fibers  at  this  site, 
the  electrolyte  (saliva)  that  allows  it  to  receive  and  maintain  electrical  contacts,  and  its  proximity 
to  the  brainstem,  minimizing  perceptual  lag.  With  the  highest  resolution  device  currently 
available,  an  approximately  25x25  tactile  image  (some  are  not  active)  is  created  by  a  sequence 
of  pulses  presented  at  a  rate  of  200  Hz.  The  amplitude  value  (voltage)  of  the  pulse  sequence  or 
'burst',  updated  at  50  Hz,  varies  with  the  grayscale  level  of  a  video  image.  The  participant 
controls  the  overall  intensity  (voltage)  level,  which  does  not  exceed  15VDC.  The  second  novel 
tactile  interface,  the  VideoTact,  is  an  electrotactile  interface  placed  on  the  abdomen  that  exploits 
the  larger  surface  area  of  the  abdomen  to  provide  a  larger  display  area.  While  the  density  of 
torso  sensory  receptors  is  lower  than  the  tongue,  placing  a  high-resolution  display  (i.e.,  24x32) 
on  the  abdomen  allows  for  rapid  perception  of  motion  of  objects  (Bach-y-Rita  et  al.,  1969)  and 
can  be  worn  discretely  under  the  user’s  clothing.  An  electrolyte  gel  is  used  to  ensure  electrical 
contact  with  the  skin. 

A  number  of  recent  and  ongoing  studies  with  the  BrainPort®  have  shown  that  blind 
individuals  can  recognize  elements  of  standard  eye  charts  (Sampaio,  Maris  &  Bach-y-Rita, 
2001,  Kupers,  Sampaio,  Moesgaard,  Gjedde  &  Ptito,  2003)  and  labyrinthine  defective 
individuals  (those  with  inner  ear  balance  disturbances)  can  dramatically  improve  balance  (Tyler, 
Danilov  &  Bach-y-Rita,  2003).  This  requires  that  the  user  have  control  of  the  movements  of  the 
sensor  in  order  to  self-generate  feedback  to  reinforce  the  novel  sensory  stimuli.  In  the  absence 
of  the  sensory-motor  loop,  we  have  observed  that  the  sensory  stimulation  through  a  tactile  array 
is  perceived  as  a  purely  tactile  experience  localized  to  the  body  part  in  contact  with  the  array. 
Laboratory  studies  and  operational  test  and  evaluation  of  the  Tactile  Situation  Awareness 
System  (TSAS)  and  other  tactile  displays  have  shown  that  individuals  rapidly  localize  and 
respond  to  encoded  tactile  cues  presented  on  the  torso  and  limbs  (Calhoun,  Draper,  Ruff  & 
Fontejon,  2002;  Diamond,  Kass,  Andrasik,  Raj  &  Rupert,  2003;  Raj,  Kass,  &  Perry,  2000; 
Sarter,  2001;  McGrath,  Estrada,  Braithwaite,  Raj  &  Rupert,  2004).  This  forms  the  rationale  for 
utilizing  self-generated  natural  movements  of  the  hands,  arms  and  body  to  control  robotic 
systems. 

Natural  motion  gesture-based  control  reduces  the  workload  and  learning  associated  with 
standard  joystick/button  input  devices  and  providing  low  latency  feedback  through  haptic 
interfaces  allow  precise  control  of  an  external  system  with  minimal  training  time.  Blinded 
servicemembers  learned  to  stabilize  and  manipulate  the  tactile  image  with  their  prosthetic  eyes 
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within  10-20  minutes,  demonstrating  that  the  tactile  feedback  does  not  need  to  be  presented 
through  the  hands  for  accurate  perception  and  control  of  a  dynamic  action.  In  year  2  of  this 
project,  we  will  evaluate  performance  of  specific  tasks  such  as  percutaneous  needle  insertion, 
for  example,  initiated  with  a  gestural  command,  and  monitored  and  adjusted  through  a  tactile  or 
auditory  presentation.  We  envision  that  one  could  switch  between  robotic  platforms  (i.e.,  dV- 
Trainer  and  Zeus®),  between  actual  patients  in  different  Trauma  Pods,  or  between  different 
surgical  instruments  (i.e.,  simulating  an  automated  tool  exchanging  mechanism)  using  gestures. 
Simple  combinations  of  movements  such  as  turning  to  face  a  different  direction  accompanied  by 
specific  hand  movements  could  provide  an  efficient  method  of  hand-off  or  change  of  procedure. 
IHMC  has  more  than  30  Zeus®  surgical  instruments.  In  this  project,  we  have  used  gesture 
recognition  agents  in  AMI  to  recognize  gestural  requests  to  lock  robot  joints  during  specific 
tasks,  such  as  changing  the  camera  angle  or  robot  arm  selection.  Further  gestures  will  be 
implemented,  such  as  performing  an  instrument  change-out  displaying  the  interpreted  request 
to  an  assistant  who  will  manually  perform  the  change,  mimicking  an  automated  tool  exchange. 

Obtaining  high  gesture  recognition 
accuracy  requires  the  ability  to  accurately 
sense  the  position  and  configuration  of  a  user’s 
hands,  limbs,  head  and  torso.  IHMC  has 
developed  custom  software  implemented  in 
AMI  that  allows  rapid,  accurate  detection  (within 
a  few  millimeters)  of  relative  positions  of  body 
components,  estimation  of  center  of  gravity, 
ground  reaction  force  center  of  pressure,  and 
estimation  of  dynamic  stability.  This  system 
incorporates  data  from  a  custom  built  wireless, 
wearable  force  sensing  insole  system  with  132 
load  cells  per  foot  (Pressure  Profile  Systems, 

Los  Angeles,  CA)  and  a  full  body/limb/head 
wireless,  wearable  motion  capture  system 
(ShapeWraplll,  Measurand,  Inc,  Fredericton, 

NB)  that  has  been  integrated  into  the  ACMI- 
SATS  prototype  to  provide  the  user  with  sense 
of  proprioception  and  interaction  with  the 
ground  through  the  haptic  interfaces  (Figure  4). 

In  addition  to  the  loss  of  haptic  and  tactile  sensation  rising  from  the  transition  from  open 
to  endoscopic  procedures,  the  surgeon  also  loses  the  significant  kinesthetic  feedback 
associated  with  gross  motor  movements.  Because  robotic  systems  (unlike  laparoscopic 
systems)  can  provide  motion  scaling  kinesthetically  rich,  large  arm  and  body  gestures  can  be 
translated  into  precise  movement  of  surgical  instruments.  Recently,  IHMC  and  its  research 
partners  demonstrated  gestural  control  of  fine  movements  of  a  robotic  arm  simulator  and 
stabilization  of  visual  imagery  represented  tactually  to  recently  blinded  servicemembers.  In  the 
former  application,  the  operator  controls  the  National  Aeronautics  and  Space  Administration 
(NASA)  Basic  Operational  Robotics  Instruction  System  (BORIS)  software  robot  (Gilliland  et  al., 
2005)  with  natural  arm  and  hand  movements.  In  the  latter,  extraocular  musculature  controls  the 
user’s  prosthetic  (glass)  eyes  which  are  rapidly  stabilized  utilizing  movement  cues  sensed  in  the 
tactually  presented  image  using  a  BrainPort®  interface  (Figure  5). 


Figure  4:  Body  position  (inset)  tracked  with 
ShapeWrap  system  using  IHMC’s  AMI 
architecture.  Box  projected  under  avatar 
indicates  stable  platform  for  a  given  stance. 
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Figure  5:  (left)  Gesture  based  controller  for  robotic  arm  simulator,  (right)  Status  post  binocular 
enucleation,  a  servicemember  uses  his  prosthetic  eyes  to  control  a  tracking  reticle  on  the  screen  with 
tactile  (tongue  based)  display  of  area  displayed  inside  the  reticle  (near  user’s  finger). 


The  first  quarter  activities  primarily  consisted  of  purchasing  required  equipment,  initiating 
development  of  a  simulation  environment  for  our  existing  surgical  robot  hardware  (Zeus, 
Computer  Motion,  w,  CA),  and  assembling  of  the  components  in  the  laboratory  space.  We 
acquired  two  additional  Aesop  arms  that  are  functionally  equivalent  to  Zeus  arms  (aside  from 
the  micro  wrist).  This  allows  us  to  implement  a  four-arm  solution  similar  to  the  da  Vinci®  system, 
and  a  spare  for  testing  hardware  modifications  (Figure  6). 


Figure  6:  Left,  Top-  IHMC  ACMI-SATS  Zeus  configuration  with  four  robot  arms.  In  the  center  of  the 
operating  table,  a  plexiglass  “patient  simulator”  provides  a  closed  environment  for  endoscopic  activities. 
Left,  Lower-  The  Zeus  can  be  controlled  by  the  stock  control  input  devices,  or  with  more  natural 
movements  using  motion  capture.  Right-  Human  scale  projected  imagery  from  the  Zeus  testbed. 
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The  virtual  prototype  of  the  Zeus  robot  was  designed  to  test  hypotheses  before 
commiting  to  physical  hardware.  The  virtual  robot  was  made  in  Java  programming  language, 
and  was  based  on  the  functionalities  of  Yobotics  Simulation  Construction  Set  (Yobotics,  Inc., 
Cincinati,  OH).  The  simulator  created  an  accessible  environment  that  allowed  for  rapid 
evaluation  and  testing  of  surgical  concepts.  The  simulation  (Figure  7)  provides  a  physics-based 
graphical  simulation  of  the  robot  arms  that  is  kinematically  accurate  and  allows  development, 
test  and  evaluation  of  ACMI-SATS  concepts  prior  to  implementation  on  the  hardware.  The 
Yobotics  SCS  allows  both  simulated  input  and  real  world  sensor  inputs  to  drive  the  motion  of  the 
arms.  Input  signals  and  robot  responses  can  be  displayed  in  stripchart  form  and  recorded. 
These  recordings  can  then  be  played  back  at  various  speeds  relative  to  real-time  to  fully  explore 
response  characteristics. 


ACMI-SATS  Zeus 


[ 


IMIMO  0  0  H 

Figure  7:  Left-  Yobotics  Simulation  Construction  Set  implementation  of  IHMC’s  current  Zeus-based 
ACMI-SATS  surgical  robot  configuration.  Top  frame  shows  graphical  model  of  configuration  shown  in 
Figure  1.  Left  panel  provides  a  list  of  joints  in  the  system  and  their  orientations,  torques,  rates,  etc.  Lower 
panel  shows  strip  chart  representation  of  variables  selected  from  the  list  in  the  left  inset  panel.  Riaht- 
Simplified  physics  only  model  of  two  robotic  arms  with  port  reaction  forces  modeled. 

The  second  quarter  entailed  integration  of  required  hardware,  continuing  development  of 
a  simulation  environment  for  our  existing  Zeus  surgical  robot  hardware  and  testing  of  the 
components  in  the  laboratory  space.  Using  AMI,  we  have  integrated  our  wireless  full-body, 
wearable  motion  capture  system  (ShapeWraplll)  with  our  Yobotics  simulation  of  the  four-arm 
Zeus  system  developed  in  the  first  quarter.  This  simulation  provides  a  physics-based  graphical 
simulation  of  the  robot  arms  that  enables  development,  test  and  evaluation  of  ACMI-SATS 
concepts  prior  to  implementation  on  the  physical  hardware.  We  updated  1970's  era  pen-plotter 
routines  for  “look  ahead”  operations  that  use  the  system  model  to  smooth  motion  despite 
varying  time  steps.  This  feature  would  be  useful  in  remote  tele-surgical  applications  with 
intermittent  communications  or  low  bandwidth  (e.g.,  TraumaPod).  Figure  7  (right)  shows  an 
intermittent  command  signal  (blue)  being  smoothed  in  to  a  smooth  velocity  using  the  look  ahead 
algorithm.  A  MIDI  control  surface  (BCF2000,  Behringer  International  GmbH,  Willich,  Germany) 
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allows  manual  positioning  of  simulation  parameters  (such  as  friction,  rates  and  joint  positions). 
This  mechanism  also  allows  tweaking  of  offsets  to  match  body  movements  to  robot  actions  in 
the  simulation.  We  have  also  completed  our  AMI  integration  of  a  10.2  channel  surround  sound 
in  the  laboratory  space.  Lastly,  our  co-investigator,  Thomas  Vassiliades,  MD,  a  minimally 
invasive  cardiovascular  surgery  expert,  traveled  to  IHMC  to  evaluate  both  the  Yobotics 
simulation  and  the  Mimic  dV-Trainer  system. 

In  the  third  quarter,  we  continued  development  of  the  simulation  environment  using 
Yobotics  Simulation  Construction  Set  for  our  existing  surgical  robot  hardware.  We  modified  the 
actual  Zeus  hardware,  and  implemented  auditory  and  tactile  interfaces  for  representation  of 
Zeus  data  using  our  Adaptive  Multiagent  Integration  (AMI)  software,  simulation  of  the  four-arm 
Zeus  system  developed  in  the  previous  quarters. 

We  also  added  head  tracking  (OptiTrack,  NaturalPoint,  Inc.,  Corvalis,  OR)  to  the  control 
input  stream  to  automatically  change  the  point  of  view  camera  location  with  user  head 
movement.  This  camera  control  mechanism  provides  motion  parallax  tied  to  the  user's  x,  y  and 
z  axis  head  movements  to  create  a  dynamic  viewpoint  that  supports  intuitive  user  interaction.  As 
the  user  moves  toward  and  away,  left  and  right,  or  up  and  down  with  respect  to  the  screen,  the 
camera  system  zooms  in  and  out  or  shifts  the  view  in  the  simulation  appropriately  to  create  an 
effective  3D  viewing  experience.  Because  drastic  movements  would  affect  system  performance, 
and  surgical  tasks  should  would  likely  be  performed  with  lower  dynamics,  simple  velocity  cut¬ 
offs  prevent  unintentional  motions  from  changing  the  view  or  moving  the  robot  We  implemented 
this  technique  with  our  Yobotics  based  simulation  in  the  third  quarter  and  then  integrated  it  with 
the  Mimic  dV-Trainer  when  we  receive  the  updated  code  in  the  fourth  quarter.  We  believe  we 
can  accomplish  the  same  effect  with  real  endoscopic  video  images  by  integrating  a  real  time 
model  creation  capability  (Tardif  &  Roy,  2005). 

In  the  fourth  quarter,  we  focused  on  integration  of  ACMI  capabilities  with  the  Mimic  dv- 
Trainer  through  our  AMI  software  using  the  same  approach  used  with  the  Zeus/Yobotics 
simulation  developed  in  the  previous  quarters.  This  includes  direct  control  of  the  Mimic  dV- 
Trainer  arm  position  and  orientation  as  well  as  closure  of  the  end  effector.  Gestural  movements 
of  the  fingers  allow  for  selection  of  mode  (e.g.,  lock  position,  clutch,  activate  electrocautery,  etc.) 
and  arm  (e.g.,  arm  two  or  three)  without  the  need  for  foot-activated  switches.  Similarly,  head 
movement-based  control  replaces  the  need  to  toggle  between  endoscope  and  arm  controls 
(figure  8).  The  aforementioned  parallax  system,  now  integrated  with  the  dv-Trainer,  provides  a 
point  of  view  that  moves  in  an  intuitive  fashion  with  the  head  such  that  moving  forward  or 
backward  increases  or  decreases  the  zoom  level,  respectively.  Vertical  and  horizontal  motion 
causes  the  point  of  view  in  the  simulation  and  changes  the  parallax  appropriately  to  provide  a 
convincing  sense  of  depth.  The  camera  can  be  locked  (using  a  gesture)  in  a  given  viewpoint  or 
moved  continuously  with  the  head  (which  provides  a  visual  sensation  similar  to  that  of  an  open 
surgical  procedure). 

We  developed  a  number  of  electrotactile  symbologies  for  representation  of  the  positions 
and  forces  on  the  end  effectors  as  well  as  relative  locations  and  forces  on  the  robot  arm  shafts. 
The  end  effector  information  displays  on  the  tongue  using  the  BrainPort®  device  and  shaft  data 
displays  on  the  abdomen  via  the  VideoTact  electrotactile  interface.  Both  the  BrainPort®  and  the 
VideoTact  devices,  as  well  as  a  10.2  channel  surround  sound  system  were  evaluated  as 
potential  sensory  augmentation  methods  for  surgical  interfaces. 
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Figure  8:  Mimic  dV-Trainer  imagery  projected  to  a  size  that  provides  a  reasonable  approximation  of  the 
human  hand  and  arm.  Control  via  motion  capture  of  hand  and  arm  movements  maps  in  a  1:1  ratio  with 
onscreen  robot  movement.  This  allows  for  veridical  kinesthetic  and  proprioceptive  cues  to  augment  visual 
information.  Screen  on  left  shows  graphical  representation  of  electrotactile  stimulus  on  the  tongue  that 
represents  both  closure  angle  (distance  between  tactile  icons)  and  forces  (intensity  of  stimulus)  on  the 
two  active  (right  and  left)  grasper  end  effectors. 


By  using  the  AMI 
framework  we  have  been  able  to 
couple  a  wireless,  wearable 
motion  capture  system,  the 
surgical  analog  environment  and 
appropriate  sensory  feedback 
displays  to  improve  the  user 
experience.  A  flow  chart  of  one 
ACMI-VAS  set-up  is  shown 
below  (Figure  9).  This  exemplar 
tracks  hand  and  digit  positions 
from  a  Measurand  AMI  software 
agent  which  is  processed  by  the 
Gesture  Grasp  Agent  to  compute 
the  distance  between  the  fingers 
of  interest  (thumbs  and  indexes 
of  each  hand).  This  value  drives 
the  selection  of  the  method  of 
intensity  level  display:  1)  Soft- 
the  intensity  first  increases 
slowly  when  the  grasper 
encounters  an  object  and  then 
more  rapidly  as  the  grasper 
closes  (similar  to  the  force 


Measurand  system: 
Gloves 


Figure  9:  Flow  chart  of  a  full  ACMI-SATS  implementation 
for  the  Zeus  suraical  robot  (see  text). 
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sensation  when  we  squeeze  a  soft  object);  2)  Medium-  the  intensity  remains  directly 
proportional  to  the  distance  between  the  fingers  (using  a  variable  gain  set  before  use);  3)  Hard- 
intensity  sharply  increases  as  soon  as  the  grasper  contacts  an  object  and  4)  Isointensity-  only 
grasper  closure  angle  is  represented,  with  intensity  held  constant. 

The  Gesture  grasp  Agent  sends  the  current  distance  between  the  fingers  and  the  current 
intensity  value  to  the  “Zeus  Gripper  BrainPort  Agent”.  The  Zeus  Gripper  BrainPort  Agent  uses 
the  values  to  create  an  image  of  the  pattern  of  stimulation,  which  is  then  sent  to  the  BrainPort 
Agent.  The  BrainPort  Agent  includes  a  graphical  user  interface  (GUI)  that  provides  visual 
feedback  to  the  researcher  regarding  the  user's  stimulus  (figure  10). 


— 
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Pattern  1  Pattern  2  Pattern  3 


Figure  11-:  Top  Row-  Three  different  symbologies  for  representing  grasper  closure  angle  on 
the  tongue.  Bottom  Row-  Two  approaches  for  end  effector  electrotactile  representation.  (Left) 
One  open  and  one  closed  grasper  (right  grasper  is  closed  firmly  resulting  in  higher  stimulus 
intensity).  (Right)  Alternate  representation  of  tactile  symbology. 

Lastly,  the  BrainPort  Agent  converts  the  image  into  a  grayscale  intensity  map  that  is 
delivered  to  the  BrainPort®  Intraoral  tongue  array  (figure  12).  We  also  prototyped  a  condition 
where  the  final  software  element  transformed  the  image  into  surround  sound  (grasper  closure 
angle  and  force  represented  by  fore/aft  sound  position  and  tone  frequency,  respectively).  In 
simulation  evaluation  activities,  the  intensities  mapped  onto  the  tongue  gave  the  user  a  finer 
degree  of  control  during  manipulation  of  the  objects  with  the  graspers.  In  future  activities 
surround  sound  will  be  assigned  to  represent  audio  sources  (e.g.,  the  sound  of  electrocautery 
activation  will  appear  auditorily  mapped  to  the  relative  location  of  the  electrocautery  end 
effector). 
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Figure  12:  Left-  BrainPort®  Intraoral  Device  electrotactile  tongue  array  (600  tactile  pixels).  Right- 
Surround  Sound  AMI  agent  that  controls  10.2  sound  system  to  place  stimuli  spatially  in  the  audio 
environment. 


To  improve  perception  of  the  locations  of  the  robot  arms  beyond  the  field  of  view  of  the 
surgeon,  we  added  symbology  to  with  the  VideoTact  to  present  the  relative  positions  of  the  end 
effector  shafts  in  screen  coordinates  (figure  13).  Stimulation  frequencies  of  1Hz,  4Hz  and  10Hz 
represented  each  arm  when  projected  onto  the  user’s  abdomen.  As  was  predicted  beforehand, 
10Hz  greatly  facilitated  recognition.  Experiments  indicated  that  three  moving  arms  were  more 
difficult  to  differentiate  effectively  with  this  symbology,  though  one  or  two  arms  were  easily 
recognized  by  a  user.  Further  activities  will  identify  optimum  stimulus  frequencies  and 
symbology  sets. 


Figure  13:  Top-  VideoTact  (ForeThought  Development,  LLC,  Blue  Mounds,  Wl)  electrotactile  abdominal 
display.  Bottom-  Graphical  representation  of  robot  arms  relative  to  endoscope  orientation.  The  tactile 
wedges  represent  the  projection  of  the  robot  arm  shafts  outside  the  field  of  view  of  the  camera  back  to  the 
entry  port.  This  allows  the  user  to  maintain  awareness  of  arm  positions,  movements  and  forces  that 
remain  outside  the  field  of  view.  The  black  rectangle  at  the  center  represents  the  camera  field  of  view 
(projected  visually). 


Approved  for  public  release;  distribution  unlimited  15 


A  connection  between  the  AMI  system  and  the  da  Vinci®  simulation  system  allowed  for 
interactive  simulation  and  testing  of  the  full  ACMI-VAS  concept.  Mimic  Technologies,  Inc., 
provided  appropriate  software  hooks  and  documentation  to  enable  AMI  to  interface  to  simulation 
variables  including  control  inputs  for  and  states  of  the  shafts  and  the  tips  of  the  instruments,  the 
positions  of  the  camera,  and  the  applied  forces  (pushing  and  pulling  forces  on  tissues  as  well  as 
grasping  forces). 

Relationship  to  award  Statement  of  Work 

At  the  end  of  the  first  year  of  the  program,  we  have  developed  a  multisensory  interface 
that  provides  visual,  audio  and  tactile  displays  as  well  as  incorporates  veridical  kinesthetic  and 
proprioceptive  cueing.  In  addition,  we  have  created  a  generalizable,  intuitive  control  interface 
for  surgical  robots  using  motion  capture  and  gesture  detection  to  command  the  robotic  system. 
Based  on  the  original  statement  of  work,  we  have  completed  the  following  tasks  from  Specific 
Aim  #  1:  Develop  a  multi-sensory  interface  to  augment  current  visual  and  haptic  interfaces  that 
will  include  of  spatial  audio  and  tactile  interfaces: 

Task  1.  Modify  IHMC  Auditory  and  Tactile  interfaces  for  tele-surgical  application 

la.  Select  tactile  displays  and  body  mounting  locations 

lb.  Define  intracorporeal  orienting  requirements 

ld.  Evaluate  tactile  and  audio  cueing  representations 

le.  Integrate  software  for  spatial  cueing 

Task  1c.  Define  registration  method  for  marking  locations  of  organs  and  vessels  was 
delayed  due  to  logistics  issues  with  the  Mimic  dV-Trainer  and  will  be  completed  early  in  year 
2.  This  will  complete  Task  1  and  produce  Milestone  #1  (Spatial  audio  and  tactile  interface 
critical  design  review). 

Task  2.  Integrate  IHMC  AMI  software  &  Mimic  dV-Trainer-Mantis  Duo 
2a.  Define  interface  over  Ethernet 

2b.  Test  transmission  of  dV-Trainer  state  variables  over  Ethernet  to  AMI 
2c.  Test  transmission  of  AMI  control  input  data  variables  to  dV-Trainer 
2e.  Develop  software  interface  to  existing  tele-surgical  robotic  end  effectors 

Task  2d.  Develop  hardware  interface  to  existing  tele-surgical  robotic  end  effectors  has  taken 
longer  than  expected  due  to  limited  documentation  on  the  Zeus  robot.  We  have  identified  all 
the  signals  that  represent  the  robot  joint  positions  and  forces  and  can  issue  control 
commands  from  an  external  computer  interface.  Integration  with  AMI  will  continue  in  year  2 
and  result  in  Milestone  #2  (Complete  software  Ethernet  interface  definition  and  testing). 

We  have  completed  the  following  tasks  from  Specific  Aim  #  2:  Develop  a  generic  tele-surgical 
robot  control  input  mechanism  that  allows  use  of  normal  kinesthetic  hand  and  arm  movements 
using  motion  capture  and  gesture  recognition 
Task  3.  Code  software  for  natural  movement  control  of  ACMI-SATS 
3a.  Define  and  integrate  motion  primitives  for  tele-surgical  activities 
3b.  Bind  coded  motion  primitives  to  dV-Trainer  control  input  commands 
3c.  Code  user  interface 

3e.  Test  and  evaluate  interface  function  with  tactile  and  audio  displays 
3g.  Test  and  evaluate  gesture  based  commands  in  dV-trainer  simulation 

Tasks  3d)  Code  performance  evaluation  interface  and  3f)  Code  finite  state  machine  (FSM) 
algorithm  for  gesture  recognition  are  progressing  and  will  be  complete  in  year  2  to  reach 
Milestone  #3  (ACMI-SATS  system  design  complete). 
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Task  4.  Demonstration  and  evaluation 

4a.  Prototype  system  functional  verification  and  testing 

Tasks  4b)  Demonstration  and  evaluation  with  subject  matter  expert  Co-Investigator  and  4c) 
Analyze  results  for  presentation  at  the  Society  of  American  Gastrointestinal  and 
Endoscopic  Surgeons  (SAGES)  annual  conference  will  occur  in  year  2  and  will  result  in 
the  Milestones  4  &  5  (Demonstrate  ACMI-SATS  system  with  simultaneous  simulated 
procedures  and  draft  manuscript  for  submission  to  Surgical  Endoscopy,  respectively) 

Key  research  accomplishments 

•  We  successfully  integrated  our  AMI  software  architecture  with  a  physical  four  arm  surgical 
robot  (Zeus)  and  with  a  commercially  available  tele-surgical  simulation  (dV-Trainer) 

o  We  created  a  physics-based  kinematically  accurate  graphical  simulation  of  the  Zeus 
robot 

•  We  integrated  spatial  audio,  tactile  and  large  format  visual  displays  for  improved  surgeon 
situation  awareness. 

o  We  have  evaluated  a  number  of  electrotactile  and  auditory  symbologies  for 
representation  of  the  positions  and  forces  on  the  end  effectors  as  well  as  relative 
locations  and  forces  on  the  robot  arm  shafts. 

•  We  have  integrated  a  wireless  full-body,  wearable  motion  capture  system  (ShapeWraplll) 
and  head  tracking  (OptiTrack)  to  allow  full  freedom  of  movement  for  the  surgeon. 

o  We  implemented  gesture  based  control  elements  into  the  ACMI-SATS  testbed  to 
provide  intuitive  mode  switching  and  tele-robotic  system  control 

•  We  implemented  a  1:1  scaling  between  arm  and  hand  movements  and  the  visual 
representation  of  the  endoscopic  view  to  ensure  useful  kinesthetic,  proprioceptive  and 
motion  parallax  cues. 

Reportable  outcomes 

We  presented  our  preliminary  approach  and  results  at  the  Aerospace  Medical  Association’s  81st 
Annual  Scientific  meeting  and  submitted  an  abstract  for  review  and  possible  inclusion  in  the 
201 1  SAGES  Annual  meeting  (see  appendix  1  for  abstracts): 

Raj,  A.  K.  &  Cameron,  J.  D.  (2010).  Multisensory  Augmentation  of  Telemedicine  Systems, 
Aerospace  Medical  Association  81st  Annual  Scientific  Meeting,  Phoenix,  AZ. 

Raj,  A.  K.,  Moucheboeuf,  A.,  Holmgren,  A.,  Hutcheson,  T.  L.,  Cameron,  J.  D.,  Lecoutre,  D.  V., 
Vassiliades,  T.  A.  (201 1 ,  submitted)  Anthro-Centric  Multisensory  Interface  for  Sensory 
Augmentation  of  Tele-Surgery.  2011  SAGES  Annual  Scientific  Meeting,  March  30-April 
2,  201 1 ,  San  Antonio,  TX. 

Conclusion 

During  year  one  of  the  ACMI-SATS  project,  we  developed  a  testbed  that  allows  for 
testing  and  evaluation  of  hypotheses  related  to  tele-surgical  robotic  human-centered  surgeon 
interfaces,  cognitive  workload,  efficiency  and  situation  awareness.  The  ACMI-SATS  testbed 
includes  both  physical  surgical  robotic  capability  (Zeus)  and  simulation  environments 
(Zeus/Yobotics,  da  Vinci®/Mimic)  to  maximize  the  efficiency  of  system  development.  We  will 
complete  the  testbed  hardware  and  software  in  year  two  of  the  project  and  complete  a  series  of 
evaluations  of  ACMI-SATS  augmentations  by  our  research  staff.  The  completed  ACMI-SATS 
testbed  could  be  used  for  future  studies  and  for  development  of  optimal  tele-surgical,  tele¬ 
mentoring  and  remote  operations  technologies  and  methodologies  for  both  military  and  civilian 
applications. 
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Appendix  1 :  Abstracts 


Raj,  A.  K.  &  Cameron,  J.  D.  (2010).  Multisensory  Augmentation  of  Telemedicine  Systems, 
Aerospace  Medical  Association  81st  Annual  Scientific  Meeting,  Phoenix,  AZ. 

Introduction:  Modern  electronics,  software  and  networking  technologies  have  enabled  the 
development  of  telemedicine  capabilities  that  bring  medical  expertise  to  remote  locations. 
Additionally,  telemedicine  enables  telementoring  to  allow  more  individuals  to  benefit  from 
interactions  with  expert  practitioners.  The  aerospace  medical  community  has  embraced  these 
novel  concepts,  which  have  the  potential  to  dramatically  improve  delivery  of  care  to  deployed 
units  or  vehicles  in  flight.  The  available  interfaces,  however,  provide  primarily  visual  information 
with  some  additional  verbal  auditory  data.  This  prevents  the  physician  from  experiencing  the 
rich  multisensory  cues  utilized  in  normal  one-on-one  patient  or  training  interactions.  Previously 
we  have  developed  an  anthro-centric  multisensory  interface  (ACMI)  system  that  can  manage 
multiple  types  of  data  streams  and  display  information  to  users  via  visual,  audio  and  tactile. 

Methods:  The  adaptive  multiagent  integration  (AMI)  software  agent  architecture  was  used  to 
integrate  video  imagery,  surround  sound  audio  and  tactile  representations  of  information  during 
telemedical  simulations.  This  ACMI  implementation  provides  typical  video  feeds  with  adjustable 
levels  of  visual  graphic  overlay.  Multiple  microphones  capture  spatial  relationships  of  the 
auditory  environment  (represented  using  surround  sound).  Combined  with  tactile  spatial 
orientations  interfaces  that  provide  perception  of  items  outside  the  visual  field  provided  by  the 
camera,  these  displays  improve  user  situation  awareness  (SA). 

Results:  Performance  (time  to  complete,  number  of  errors)  improved  for  individuals  when  using 
telemedicine  apparatus  with  ACMI  vs.  no  ACMI  enabled  for  a  series  of  telemedicine  relevant 
simulated  tasks. 

Discussion:  Telemedicine  holds  great  promise  for  use  in  aerospace  environments. 
Augmentation  with  multisensory  displays  could  enhance  effectively  and  expand  capabilities  and 
applications  without  increasing  user  workload. 

Learning  Objective  1 

Current  methods  for  multisensory  interface  implementations  will  be  described. 

Learning  Objective  2 

Techniques  for  improving  multisensory  situation  awareness  without  increasing  system 
bandwidth  usage  will  be  discussed. 

Learning  Objective  3 

Effectiveness  of  multisensory  interface  in  telemedicine  applications  will  be  discussed. 
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Raj,  A.  K.,  Moucheboeuf,  A.,  Holmgren,  A.,  Hutcheson,  T.  L.,  Cameron,  J.  D.,  Lecoutre,  D.  V., 
Vassiliades,  T.  A.  (2011,  submitted)  Anthro-Centric  Multisensory  Interface  for  Sensory 
Augmentation  of  Tele-Surgery.  2011  SAGES  Annual  Scientific  Meeting,  March  30-April 
2,  201 1 ,  San  Antonio,  TX. 

Objective  of  Technique:  While  telerobotic  surgical  systems  augment  minimally  invasive 
surgery  techniques  and  reduce  surgical  trauma,  they  increase  surgeon  workload  by  limiting 
sensory  feedback.  Unlike  open  or  laparoscopic  procedures,  surgical  robots  isolate  the  surgeon 
from  tactile,  proprioceptive,  kinesthetic  and  orientation  cues,  which  provide  non-visual  inputs 
that  help  maintain  situation  awareness  (SA).  The  robot  control  system,  however,  employs 
sensors  for  torque,  position,  velocity  and/or  strain  sensors  to  maintain  accurate  closed-loop 
servomotion,  which  could  help  inform  the  surgeon's  SA.  Though  sitting  at  a  console  compares 
favorably  to  standing  astride  an  operating  table  using  laparoscopic  instruments,  removing  all 
restrictions  on  surgeon  motion  and  posture  would  allow  telerobotic  surgeons  to  move  freely  and 
reduce  both  physical  and  cognitive  fatigue.  Lastly,  the  surgeon  must  change  modes  between 
control  of  primary  instruments  and  any  additional  arms  (e.g.,  third  effector,  endoscope,  etc.),  but 
current  robot  interfaces  provide  little,  if  any,  feedback  to  help  the  surgeon  maintain  awareness 
of  such  changes. 

The  objective  of  the  Anthro-Centric  Multisensory  Interface  for  Sensory  Augmentation  of  Tele- 
Surgery  (ACMI-SATS)  project  seeks  to  restore  to  telerobotic  surgery,  many  of  the  sensory  and 
kinesthetic  cues  available  in  open  surgical  procedures.  This  improves  the  effectiveness  of  tele¬ 
surgery  by  allowing  a  surgeon  to  utilize  tactile,  spatial  audio  and  three-dimensional  visual  cues 
as  well  as  meaningful  proprioceptive  and  kinesthetic  information 

Description  of  the  methods:  The  Florida  Institute  for  Human  and  Machine  Cognition  (IHMC) 
developed  the  ACMI-SATS  system  with  both  simulated  and  actual  surgical  robot  systems.  It 
provides  an  integrated  architecture  with  a  wide  field  of  view  pseudo-three  dimensional  visual 
operative  field,  surround  sound  for  spatially  relevant  audio  when  selecting  instruments  and 
multiple  tactile  interfaces  that  represent  instrument  dynamics.  ACMI-SATS  provides  a  natural 
free  motion  control  interface  that  uses  a  wearable,  wireless  motion-capture  system  to  track 
head,  torso,  arm,  wrist  and  finger  movements.  Natural  motions  drive  the  movement  of  the 
robotic  instruments  and  the  endoscope.  The  surgeon  can  stand  or  sit  without  any  external 
restrictions  and  proprioceptive  and  kinesthetic  sensations  map  to  the  visual  presentation  at  a 
1:1  scale.  Motion  capture  enables  gestural  control  to  manage  mode  changes  (such  as  clutching 
or  switching  control  to  a  different  arm). 

Preliminary  results:  Initial  evaluations  on  the  ACMI-SATS  testbed  indicate  that  novices  (non¬ 
surgeons)  can  learn  to  control  and  manipulate  the  robotic  end  effectors  to  perform  minimally 
invasive  surgical  training  tasks  in  simulation  quickly  and  with  low  cognitive  effort. 

Conclusions/Expectations:  The  sensory  interfaces  augment  understanding  by  providing 
additional  information  to  the  surgeon  intuitively  without  overloading  visual  or  auditory  sensory 
channels.  The  motion-capture  and  gestural  control  interface  allows  the  surgeon  to  use  more 
natural,  unrestricted  movements  control  robot  actions  and  mode  changes.  By  mapping  the 
visual  scale  directly  to  motion  of  the  arms,  the  surgeon  can  maintain  awareness  of  instrument 
positons  even  when  they  are  no  longer  with  the  endoscope  field  of  view.  ACMI-SATS  can  be 
integrated  with  any  surgical  robot  platform  and  could  allow  surgeons  to  learn  and  perform  more 
procedures  in  a  given  time  frame,  with  less  effort  and  with  fewer  surgical  errors. 
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Appendix  2:  Acronyms 


ACMI-SATS  -  Anthro-Centric  Multisensory  Interface  for  Sensory  Augmentation  of  Tele-Surgery 
AMI  -  Adaptive  Multiagent  Integration 
BMI  -  Brain-Machine  Interface 

BORIS  -  Basic  Operational  Robotics  Instruction  System 

DARPA  -  Defense  Advanced  Research  Projects  Agency 

DoD  -  Department  of  Defense 

ERF  -  Electro-rheological  fluid 

fMRI  -  Functional  Magnetic  Resonance  Imaging 

FSM  -  Finite  State  Machine 

GUI-  Graphical  User  Interface 

HAPsMRT  -  High  Altitude  Platforms  Mobile  Robotic  Telesurgery 

HD  -  High  Definition 

HMD  -  Head  Mounted  Display 

HMI-  Human  Machine  Interface 

IED  -  Improvised  Explosive  Device 

IHMC  -  Institute  for  Human  and  Machine  Cognition 

NASA  -  National  Aeronautics  Space  Administration 

SA  -  Situation  Awareness 

SAGES  -  Society  of  American  Gastrointestinal  and  Endoscopic  Surgeons 
TSAS  -  Tactile  Situation  Awareness  System 
UAV  -  Unmanned  Aerial  Vehicle 
VE  -  Virtual  Environment 
VR  -  Virtual  Reality 
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